DOCUMENT RESUME 



ED 395 Oil 



TM 025 023 



AUTHOR 

TITLE 



INSTITUTION 
SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Meehan, Merrill L.J Wiersma, William 
Impact of Appalachia Educational Laboratory's 
Products and Services on Its Clients in Three 
Employer Groups and Five Role Groups. 

Appalachia Educational Lab., Charleston, W. Va. 
Office of Educational Research and Improvement (ED), 
Washington, DC. 

Jun 95 

RP91002002 

53p. 

Reports - Evaluative/Feasibility (142) — 
Tests/Evaluation Instruments (160) — Statistical 
Data (110) 

MF01/PC03 Plus Postage. 

Administrators; *Educat ional Research; Elementary 
Secondary Education; Employers; Evaluation 
Utilization; Federal Aid; Financial Support; Higher 
Education; Information Dissemination; Profiles; 
Program Evaluation; Questionnaires; ^Regional 
Laboratories; *Research Utilization; ^Teachers; User 
Satisfaction (information); Use Studies 
* Appalachia Educational Laboratory WV; Vf Impact 
Evaluation 



ABSTRACT 

As one of the federally-funded regional educational 
laboratories, the Appalachia Educational Laboratory LAEL) is 
committed to assessing the impact of its products and services on a 
variety of client subgroups. These include educators and 
administrators at the elementary, secondary, and higher levels. To 
address the issue of measuring impact, the AEL identified seven 
impact categories: (1) overall satisfaction; (2) knowledge and 
skills; (3) information sharing; (4) use in planning; (5) use in 
implementation; (6) indirect benefits; and (7) secondary clients' use 
in implementation. Over a 3-year period, the AEL measured impact with 
a client questionnaire consisting, in the final version, of 5“item 
subscales that were administered to 672 individuals in primary client 
subgroups. Scores from this normative group were converted to 
percentiles, and mean scores were plotted as profiles for subgroups. 
Results then reflected specific professional roles and the impacts of 
the AEL on each. An appendix presents the AEL questionnaire 
development checklist. (Contains 3 tables, 10 figures, and 12 
references . ) (SLD) 
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The Appalachia Educational Laboratory (AEL), Inc., works with educators in ongoing R & D- 
based efforts to improve education and educational opportunity. AEL serves as the Regional 
Educational Laboratory for Kentucky, Tennessee, Virginia, and West Virginia and operates the 
Eisenhower Math /Science Consortium for these same four states. It also operates the ERIC 
Clearinghouse on Rural Education and Small Schools. 

AEL works to improve: 

• professional quality, 

• curriculum and instruction, 

• community support, and 

• opportunity for access to quality education by all children. 

Information about AEL projects, programs, and services is available by writing or calling AEL. 



This publication is based on work sponsored wholly or in part by the Office of Educational 
Research and Improvement, U. S. Department of Education, under contract number RP91002002. 
Its contents do not necessarily reflect the views of OERI, the Department, or any other agency of the 
U. S. Government. 




Post Office Box 1348 
Charleston, West Virginia 25325-1348 
304/347-0400 
800/624-9120 (toll-free) 
304/347-0487 (FAX) 



AEL is an Affirmative Action/Equal Opportunity Employer. 
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EXECUTIVE SUMMARY 



As one of the federally-funded Regional Educational Laboratories, the Appalachia Educational 
Laboratory (AEL) is committed to assessing the impact of its products and services upon the clients 
it serves. This task raises two issues: (1) the measurement of impact and (2) the nature and extent 
of impact among the subgroups of the client population. With regard to the latter, AEL serves a 
variety of client suogroups, most in education-related professional roles. These subgroups include 
teachers, principals, central office staff of school systems, state education agency personnel, and 

higher education professors and administrators. 

To address the issue of measuring impact, AEL identified seven impact categories which, to 
some extent, were an adaptation of the “Continuum of Impact” stages identified by the Northwest 
Regional Educational Laboratory. The seven categories that are somewhat hierarchial include: 
Overall Satisfaction, Knowledge and/or Skills, Information Sharing, Use -in Planning, Use in 
Implementation, Indirect Benefits, and Secondary Clients’ Use in Implementation. These impact 
categories seem to provide adequate coverage of the types of impact intended by AEL. 

Over a three-year period (1992-1994), AEL measured impact with a 50-item client 
questionnaire containing seven items for each impact category, plus a veracity-check item. This 
generated a substantial database. The number of items per impact category was reduced to five, 
eliminating two items from each category subscale and retaining the five items that maintained the 
highest internal consistency reliabilities. As a result, the five-item subscales were highly reliable (all 
r’s .92 or greater). A total of 672 individuals had completed data on all subscales and these 
individuals represented the primary client subgroups served by AEL. The norming process was based 
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on the data of these 672 questionnaire respondents, and this group of respondents was the normative 
group. 

Scores of the normative group were converted to percentiles, and the mean scores on the 
subscales were plotted as profiles on the percentile scale. Respondents were identified in terms of 
their employers and professional roles. The latter gave a more detailed breakdown because it 
separated teachers, principals, and central office staff in school systems; and professors and 
administrators in colleges and universities. The former was useful for comparisons involving state 
education agency personnel. Individual and composite subgroup profiles were generated. 

One of the most significant results was the overall magnitude of AEL’s impact. Even with 
those subgroups impacted the least, the impact was substantial. 

The comparison of the subgroups of the client population showed substantial differences in 
the profiles. Educational leaders such as principals, central office staff, and state education 
agency personnel reported high impact overall and especially on the planning, use in implementation, 
and secondary clients’ use in implementation subscales. Teachers were somewhat low in planning. 
Higher education personnel tended to have the lowest positions, especially the administrators. These 
results are a reflection of the specific professional roles and, to some extent, the program emphasis 
ofAEL. 

The norming process has proved valuable in determining AEL’s impact as it may vary across 
the client subgroups. The profiles based on the normative data reveal the nature of AEL impact 
across subgroups. The 35-item questionnaire was found to generate a useful and credible database. 
The results of the norming process provide information for summative evaluation, as well as data for 
formative evaluation of AEL’s programs, products, and services. 







INTRODUCTION 



The Appalachia Educational Laboratory (AEL) is a private, nonprofit corporation with 
headquarters in Charleston, West Virginia. AEL was incorporated in 1966. The Laboratory was 
designed and established by educators committed to improving education and educational opportunity 
in its four-state Region-Kentucky, Tennessee, Virginia, and West Virginia. The Lab is governed by 
a 28-member Board of Directors representing various stakeholders of each state, including state 
departments of education, higher education institutions, classroom teacher associations, school 
administrator associations, and the public at large. In fact, 54% of the membership of the AEL Board 
of Directors consists of representatives of the general public. 

AEL’s work falls into four categories: (1) the Regional Educational Laboratory (REL), 
(2) the ERIC Clearinghouse on Rural Education and Small Schools; (3) the Eisenhower 
Mathematics/Science Consortium; and (4) small contracts with local, state, and regional 
organizations. The REL, ERIC Clearinghouse, and Math/Science Consortium are funded under 
separate contracts with the Office of Educational Research and Improvement (OERI), U S. 
Department of Education. Twenty RELs were created in 1966 and only six of those original Labs 
remain in operation. AEL is one of those six original Labs. Thus, longevity is one indicator of AEL’s 
success. But, there must be other indicators of Lab impact in its Region. 

How does a federally-funded, multistate, educational research and development organization 
assess itself and report on its effectiveness? How does an organization, charged with improving 
education in its multistate Region by linking the world of research with the world of practice, evaluate 
itself? Over the years, AEL has employed several alternative methods to assess and repoi l on its 
impact on education in its Region. These methods include constituent surveys, tracer case studies. 
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retrospectives on important events, intensive case studies, and in-depth interviews with key 
informants. Each of these evaluation methods yielded credible and important new information; 
however, there were few common understandings across the evaluation outcomes of the studies using 
the various methods. There were no common terms to help define and explain the various outcomes 
of the many evaluation efforts. 

AEL’s search for assessing and describing the impact of its work led to the problem of 
categorizing the various types of activity impact and using those categories to assess and report on 
efforts to itself, to its Board of Directors, to its funding agency, and to others. 

AEL Impact Categories 

In its contract to operate the REL for the Appalachian Region in the 1991-95 period, AEL 
is obligated to document, describe, and report on the “forms” of impact it made in the Region. AEL 
contracted to define the impact of its work and to develop different categories of impact. The 
purpose for different categories of impact was to provide terms that differentiated the various types 
of impact made by the individual programs within AEL and, also, the various types of activities 
program staff conduct to complete their work. Another purpose was to have a means by which the 
impact data could be aggregated to cross-program and institution levels. 

Prior to AEL’s work on impact, evaluators at the Northwest Regional Educational Laboratory 
(NWREL) developed a “Continuum of Impact” to describe their evaluation outcomes. NWREL 
evaluators described their continuum as consisting of six stages of impact (Gabriel, 1988). Their 
1988 self-assessment report (Gabriel, 1989) concluded that their stages of impact were useful both 
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for describing aspirations of outcomes before data collection and for categorizing the results of data 
collection. 

In 1990, AEL evaluators conducted a two-phase project to validate the NWREL stages of 
impact with extant interview data from AEL clients and then to produce AEL s definition and 
categories of impact. In the first phase of the validation project, evaluators at Western Michigan 
University (WMU), AEL’s external evaluator, tested the six NWREL stages of impact on a set of 64 
transcribed interviews from key AEL clients. Four WMU evaluators independently read the interview 
transcripts and assigned the recorded outcome data to one or more of the six stages of impact. With 
94% overall intrarater agreement, the WMU staff concluded that the NWREL stages were valid for 
AEL’s client data with two adjustments. The WMU evaluators concluded that “planning” should be 
redefined and that the NWREL planning definition lends itself to a new stage/category of 
“information sharing” (Meehan, 1992). The second conclusion from the WMU evaluators was that 
the separate impact categories do not fit neatly into a continuum. The WMU report strongly urged 
AEL staff not to represent or to present the impact categories as interrelated stages, levels, or steps 
in a continuum (Seita, Vogel, Sanders, & Lacar, 1990). 

In .the second phase of the 1990 study, AEL staff moved the impact categories effort forward. 
AEL evaluators developed a unique definition of impact for its work with clients. Building on the 
WMU conclusions, AEL staff added a new category on information sharing to the previous list. 
Next, AEL staff developed new names for some of the NWREL categories. Then, operational 
definitions for the seven categories were developed to add precision and clarity. Next, these three 
items (general definition of impact, seven categories of impact, and the operational definition for each 
category) were submitted for internal review and revision. The completed review and revision 



process showed the utility and potential of these essential components of impact for AEL evaluation 
activities. Table 1 displays the AEL impact definition, category names, and the operational definition 
for each category (Meehan, 1991, 1992). 

Pilot Test of Instrument 

In 1991, AEL evaluators completed a pilot test of an instrument based on the seven categories 
ofimpact. The purposes of the pilot test were (1) to develop and administer a pilot-test version of 
an instrument to measure the impact on various client groups in each impact category, (2) to compute 
and present the results of the reliability estimates for all seven subscales and the total instrument, and 
(3) to draw conclusions from the pilot test of the instrument to measure institutional impact and to 
present recommendations for next steps (Meehan & Wiersma, 1993). 

The development of the pilot-test instrument followed several stages between March and 
October 1991. First, draft items were produced individually by the AEL program managers for the 
seven impact categories, based on one sample item per category written by the AEL internal 
evaluator. All these draft items were compiled into lists by categories for use in a working meeting. 
Next, during this working meeting of the program managers, the draft items were read, reviewed, 
discussed, and debated as to their fitness/utility as pilot-test items. Duplications in the items were 
eliminated and refinements were made. The group chose the five-option Likert scale response format. 
The response options ranged from “Not at all” to “Much, ’ “Many, or Very . Then the internal 
evaluator summarized the group’s work by compiling the 29 agreed-upon refined items into the pilot- 
test instrument. The number of items per impact category ranged from a low of three to a high of 
seven. The items were kept in the order of categories from the original AEL information sheet. 
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Table 1 



Appalachia Educational Laboratory 
IMPACT - Definitions and Categories 

Impact : At AEL, impact is defined as changes in clients’ attitudes, knowledge, practices, or policies. 
Further, at AEL, these impact changes are organized into seven different categories. 



Category 


Definition 


1 . Overall Satisfaction 


AFT, primary clients view the content and presentation of AEL 
products and/or services as desirable, useful, or effective. 


2. Knowledge and/or Skills 


AEL primary clients gain new knowledge or skills as a result 
of AEL products/services or renew previously learned 
knowledge or skills. 


3. Information Sharing 


AEL primary clients share the information, materials, and/or 
skills from their contact with AEL with their colleagues in 
their own environment or with others in other environments. 


4. Use in Planning 


AEL primary clients use information, materials, and/or skills 
gained from AEL in developing plans that include, for 
example, program design, goals, objectives, activities, 
timelines, outcomes, and evaluation. 


5. Use in Implementation 


AEL primary clients change their job-related 
practices/behaviors in their own environments as a result of 
the products or services delivered by AEL. 


6. Indirect Benefits 


AEL primary clients report unanticipated or serendipitous 
outcomes related to the AEL products or services. 


7. Secondary Clients’ Use 
in Implementation 


AEL secondary clients (the clients of AEL’s primary clients) 
change their attitudes, knowledge, behaviors, or practices in 
their own environments based on the products or services 
delivered by AEL. 



The sample for the pilot-test version of the impact instrument was determined by AEL program 
managers because it was one part of a dual-purpose annual survey for which the first section was 
program-specific. Thus, AEL program managers decided what client group was to receive their 
part 1 survey and these groups then made up the sample for part 2, the impact instrument. The total 
number of AEL clients who received the pilot-test impact instrument in 1991 was 417. There were 
261 instruments returned with usable data. Of those, 253 answered the three demographic questions. 
The number of respondents per state was Kentucky, N = 61 (24.1%); Tennessee, N = 56 (22.1%); 
Virginia, N = 76 (30.0%); and West Virginia, N = 53 (20.9%). Seven respondents checked “Other.” 
Nearly 80% of the respondents were in just two of the seven employer categories: local education 
agency (43.1%) and institutions of higher education (36.0%). Regarding professional role, 20.9% 
were teachers, 17.8% were higher education professors, 16.2% were higher education administrators, 
13.0% were central office staff, and 9.1% were principals or assistant principals. The four remaining 
categories had less than 4% each in them. 

The internal consistency reliability of the pilot-test instrument was measured by Cronbach’s 
Alpha. Of the 261 completed instruments with some usable data, there were 151 with responses to 
the full set of 29 items. The Alpha reliability coefficient for the full instrument was a very high .97. 
The Alphas for the seven different impact categories ranged from a low of .71 for one scale (of four 
items) up to a high .94 for another scale (of seven items). Three of the impact category Alphas were 
in the .70s, one was in the .80s, and three were in the .90s. Predictably, the Alphas increased as the 
number of items in the category subscale increased. (Adams, Barley, Gullickson, Keller, & Wiersma, 

1992). 
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Due to the different number of items for each of the seven categories, item means and standard 
deviations were used for interpretation purposes. The possible responses were from 1 to 5 (low to 
high). The seven subscales for the categories produced item means that ranged from 3.37 for 
category #7 to a high of 4.30 for category #1 . The most obvious and interesting trend in the item 
means by subscales was their small but steady decline from the first category (Overall Satisfaction) 
to the seventh category (Secondary Clients’ Use in Implementation). Yet, even with this decline, all 
the item means were in the positive portion of the scale. The decline in subscale means is 
understandable. Items in the first three impact categories are directly controllable by the individual 
respondent, while items in the last four categories deal with the respondent interacting with other 
individuals and much less direct control. 

Development of the Field-Test Instrument 

One of the recommendations from the pilot test of the 29-item version of the impact instrument 
was to expand the number of items for each category. A second recommendation was to equalize 
the number of items for each category in the instrument. The purposes of the first recommendation 
were to increase the number of viable items available for each category and to raise the reliability 
estimates for some subscales. The purpose of the second recommendation was to provide a common 
scale for the scores of the seven subscales, thus enhancing direct comparisons. 

Because one category with seven items produced an Alpha reliability estimate of .94, that 
number of items was selected as reasonable for the six other categories. The AEL internal evaluator 
wrote 20 new items for the field-test version so that each category had seven items. The field-test 
instrument had 49 items at this stage (7x7 = 49), so it was decided to add one more item to have 
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the round number of 50 items total. Given this opportunity to add one additional item into the field- 
test impact instrument, a veracity-check item was written. That is, one item with no mention of AEL 
in it was added as the 50th item to assess if respondents really did read the stem statements. 

The 20 new items were reviewed by several AEL program managers for content. The veracity- 
check item was not part of this review process. Minor word changes in a few items were suggested 
by the AEL program managers, and these improvements were made. After the 21 new items 
(including the veracity-check item) were- finalized, they were assembled into the field-test version of 
the impact instrument. As in the pilot test, the items in the seven categories were assembled in the 
order of the categories on the original AEL information sheet, category #1 through category #7. That 
is, the 50 items in the field-test impact instrument were ncl put in random order; they were in the 
order of the seven impact categories. The three demographic questions were retained from the pilot- 
test version. The development and two administrations of the 50-item field test impact instrument 
was reported by Meehan and Wiersma (1994). 

Purpose and Objectives of This Study 

The major purpose of this study was to combine the data sets resulting from three annual 
administrations of the AEL impact instrument to a sample of its clients, to establish the norms for the 
combined database, then to profile and describe the impact of AEL on its clients in three major 
employer groups and five major role groups. 

The objectives of this study were (1) to combine the data sets from three annual administrations 
of the AEL impact instrument, (2) to reduce the length of the instrument from 50 items to 35 items 
and assess the change in the scale reliabilities, (3) to establish the norms for the seven subscales and 
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the total instrument based on those clients who responded to all 35 items, (4) to graph the impact 
profiles for clients by three major employer groups and five major role groups, and (5) to analyze the 
resulting impact graphs for AEL clients and draw conclusions from those analyses. 



INSTRUMENT ADMINISTRATION AND REDUCTION 



This section discusses the three samples for the AEL impact instruments, its administration 
procedures, the reduction of the instrument from 50 to 35 items, and the Alpha reliabilities for the 
subscales and total instrument. 



Three Annual Samples 

Field test #1. The first field test of the expanded impact instrument was completed in 1992. 
There were 365 impact instruments mailed and 243 usable responses received. As expected, not all 
the respondents answered all 50 items and the three demographic questions. West Virginia had the 
largest number of respondents (34.9%), while Tennessee had the smallest number (10.7%). Kentucky 
and Virginia were only one respondent apart; each had a little more than 25% of the total. A few 
respondents (N = 7) marked “Other” or did not respond to this item. 

In terms of the employer of the respondent, the largest number marked (N = 129) was the local 
education agency at 53.1%. The next most frequently marked employer was the state education 
agency at 14.4%, followed by institution of higher education at 8.2%. Community organization was 
the next category at 5.8%. Policy-oriented organization and professional education association 
employer categories each had less than 3% of the respondents, while two national/federal government 
categories yielded no respondents. Interestingly, 13.2% marked the ‘ Other category or did not 
respond to the question. 

Regarding the professional role marked by the respondents, the largest number (N = 92 or 
40%) indicated the teacher category. The second most marked role group was central office staff at 
11.1%, followed by principal/assistant principal at 6.6%. Eight different role groups each had less 
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than 5% of the respondents marking them. They included higher education professor (4.9%), 
community organization (4.5%), state board of education and key staff (1.2%), higher education 
administrator (3.7%), state legislator or staff (1.6%), chief state school officer or staff (1.2%), 
governor and/or key staff (0.8%), and local board of education member (0.4%). Apparently the 1 1 
role groups were not enough, as over one fifth (20:5%) of the respondents marked the “Other 
category. Finally, 2.1% did not respond to this question (Wiersma, 1993; Meehan & Wiersma, 

1994). 

Field test #2. The second field test of the 50-item impact instrument was completed in 1993 . 
Of the 447 AEL clients sampled, 271 usable instruments were returned and 8 others were not 
delivered or unanswered for other reasons. The largest number of respondents (32.5%) were 
employed in Virginia and the smallest number (1 5. 1%) were employed in Kentucky. In the middle, 
26.6% were employed in West Virginia and 21.8% were employed in Tennessee. 

Over two thirds of the respondents (71 .6%) were employed by local education agencies. The 
second largest number of respondents (10.7%) were employed by institutions of higher education. 
After these two types of employers, the percentage of respondents dropped markedly with no 
employer having more than 6%. Of those, though, community organizations had the most 
respondents. Thirteen respondents marked “Other,” while 10 did not respond to this question, for 

a combined 8.5%. 

Of the respondents, 46.1% said they were teachers, the professional role marked most 
frequently. The second most frequently checked role was principal or assistant principal (39 for 
14.4%), followed by 28 central office staff (10.3%). Those employed by institutions of higher 
education were split almost evenly between professors and administrators, 15% and 13% respectively. 
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After the local education agency and institution of higher education employer-type professional role 
categories, the role group responses dropped off markedly with less than 2% for any specific role. 
As was the case in both prior administrations of the impact instrument, the professional role 
categories provided in the list seemed not to be applicable for many of the respondents. In this field 
test, 35 respondents (12.9%) marked the “Other” category or did not respond to the question 
(Wiersma, 1 994; Meehan & Wiersma, 1 994). 

Field test #3. The third field test of the 50-item impact instrument was completed in 1994. 
This was the largest sample of AEL clients to receive the impact instrument. The 1994 sample 
consisted of 618 AEL clients distributed across eight AEL programs, representing 10 samples or 
populations selected by the program managers. A subset of 28 instruments, letters, or followup 
postcards were returned as undeliverable or unable to respond for some legitimate reason. This 
reduced the effective mailing to 590 AEL clients. There were 338 usable impact instruments returned 
for an effective response rate of 57.3%. 

As usual, not all respondents answered all the impact items nor all of the demographic items. 
A total of 308 indicated the state in which they work. The respondents were rather evenly distributed 
across AEL’s four states: Virginia had 71 (23.1%), Tennessee had 70 (22.7%), Kentucky had 66 
(21 .4%), and West Virginia had 65 (21 . 1%). Thirty-six respondents (1 1 .7%) checked the “Other” 
category, indicating they worked in states not in AEL’s four-state Region. This was due in part to 
one of the programs conducting workshops outside of the Region. 

Of the 306 respondents who responded to the employer category, 1 55 (50.6%) marked the 
local education agency. The next highest employer category marked was institution of higher 
education with 30.1%. With these two employers having 80% of the respondents, the remaining 
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employer categories had small numbers and percentages. Of those, 4.9% checked the “Other” 
category and 4.8% marked the state education agency. The professional education association 
employer category was marked by 3.3% of the 1994 respondents and the remaining categories all had 
smaller percentages of respondents. 

Regarding the role group checked by respondents, the largest percentage was teachers at 
34.9%. This is consistent with previous administrations. More respondents were employed in the 
central office staff (25 for 8.1%) in a local education agency than as principals or assistant principals 
(17 for 5.5%). Fifty-two respondents (16.9%) were college or university administrators and 36 
(1 1 .7%) were employed as college or university professors. Following these professional roles, the 
percentages for the remaining roles were small at less than 3% for any role (Wiersma, 1995). 

Instrument Administration 

The administration of the expanded impact instrument was similar for the three field tests 
(1992, 1993, and 1994); therefore, a generic explanation of the process is presented below. 

The 50-item field-test instrument to measure the impact of an R & D organization was 
administered as part of the annual survey of AEL clients. The first part, called Section 1 in the 
completed survey, was designed by program managers specifically for selected subgroups of their 
clients. These subgroups were selected because they had received specific products or services from 
the particular program during the year or prior years. These subgroups of clients were selected 
mainly as purposeful samples but, in several cases, were drawn at random. 

The field-test impact instrument was the second part of the annual client survey. These 
second sections were exactly alike for all AEL clients, except they were printed on different colored 
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paper for easy identification purposes. Section 2 of the annual client survey was titled “General 
Perceptions of AEL Services and Materials” and it contained a proper copyright notice. 

AEL and WMU staff collaborated on the administration of the client surveys, including the 
impact instrument (Section 2). Cover letters were prepared at WMU, on The Evaluation Center 
letterhead, and then were reviewed and approved by AEL program managers. All of the mailings of 
the surveys originated at AEL, but all completed surveys were sent directly to WMU for analysis. 

There were four different waves of mailings for the client survey, including the field-test 
impact instrument. Initial mailings of the full survey packages were completed in the third week of 
October. Each survey package included the cover letter from WMU evaluators; the survey consisting 
of two sections; and a postage-£aid, self-addressed return envelope for the completed survey. A code 
number was written on each instrument for tracking purposes. WMU staff checked off each returned 
survey as it was received at their offices. Then, WMU staff faxed a list of nonrespondents to AEL 
coffin early November. AEL staff prepared and mailed a first followup postcard reminder to those 
nonrespondents. WMU staff continuously tracked the returned surveys and faxed a second list of 
nonrespondents to AEL staff in early December. At this time, a second complete survey packet was 
sent to all remaining nonrespondents with a letter from WMU urging completion and return of the 
instrument. The last mailing consisted of a final appeal to nonrespondents in the form of a message 
typed on a bright orange label that was affixed to a postcard. Receipt of the surveys was closed 
immediately after the holiday vacation period at the WMU campus. In 1994, a prenotification letter 
was sent as the first mailing in an attempt to increase the response rate. However, this step was 
judged to be ineffective since the response rate did not improve. 
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The return rates declined slightly over the three field-test years. In 1992, 365 instruments 
were mailed to the sample and 243 were returned by the mid-December analysis cutoff date for a 
66.9% return rate. However, a group of 39 instruments was received later and added into the total 
database. A substudy showed no significant difference in the responses from the “late returners” 
when compared to those of the “on time” returners. This made the return rate of usable instruments 
in 1992 of 77.5%. In 1993, 447 instruments were mailed initially and 271 usable responses were 
received for a return rate of 60.6%. However, eight surveys were returned as undeliverable or not 
able to be completed for other reasons. So, the response rate of usable instruments moved up to 
61.7%. For 1994, the initial mailing consisted of 618 impact instruments. However, a subset of 28 
was returned as undeliverable or with notes explaining why the respondent could not complete the 
instrument, such as no longer employed in that professional role. There were 338 usable instruments 
for a return rate of 57.3%. This figure was close to the 1993 return rate, but almost 10 percentage 
points less than the initial return rate for 1992. 

Instrument Reduction 

With three annual administrations of the AEL impact instrument completed, there was a 
sufficient number of completed versions to aggregate them into one data set and assess the impact 
of reducing the 50 items to a more convenient number. The purpose of reducing the number of items 
would be to shorten the amount of time required for respondents to complete the instrument, possibly 
improving the return rate. Of course, the goal at this stage was to reduce the number of items per 
subscale, only if the Alpha reliabilities were not adversely affected (i.e., substantially lowered). 
Another reason for assessing the process of reducing the impact instrument in 1995 was tha. 
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sufficiently diverse types of AEL clients had completed the instrument over the three years to warrant 
further investigation of the instrument reliabilities. That is, it took three years to sample all the 
various types of AEL clients from the nine different programs. There were empty cells or low 
numbers in some of the employer and/or role group cells for the first two years. The large sample 
in the 1994 administration rounded out most of the categories that had low numbers. One exception, 
though, was the “I attended as a parent” category in both the employer and role groups, which was 
a new addition in 1 994 based on input from AEL program managers. 

After combining the completed AEL impact instruments from the 1992, 1993, and 1994 
administrations, the database contained 822 cases, all of which had completed a minimum of 26 of 
the 50 instrument items. Since the response scale for all ite* v ' was a low of 1 (Not at all) to a high 
of 5 (Much, Very, or Many), the decision was then made to include only those respondents who had 
actually completed every item in the instrument. The position taken was that omitting an item would 
yield a 0 value for that item, but a 0 was not one of the actual response options. Thus, only those 
respondents who chose one of the actual response options should be included in the instrument 
reduction phase. Estimating reliability using the Cronbach Alpha coefficient also requires complete 
data. 

When those respondents who omitted one or more items of the instrument were dropped from . 
the database, the number remaining was 672. Therefore, the database for reducing the instrument 
consisted of 672 AEL clients who had completed all impact items in 1992, 1993, or 1994. There 
were no duplicate names within any one year’s administration because a specific step was taken in 
assembling the mailing list each year to eliminate possible duplicates. 
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Actual reduction of the 50 items (including one veracity-check item) to 35 items was rather 
straightforward. First, the Alpha reliability runs were made for the seven subscales and for the total 
instalment. As part of the regular SPSS-PC+ computer printouts for the Alpha reliabilities, the value 
of Alpha for the subscale or total score if every individual item was omitted is provided. Thus, it is 
possible to assess the contribution of each item to the total Alpha reliability. The goal of reducing 
each subscale by two items was selected. Reducing each subscale by two items would yield a total 
instrument of 35 items (7x5 = 35), since the veracity-check item also was eliminated. 

The second step in reducing the impact instrument was to inspect each item’s contribution to 
the Alpha reliability coefficient for each category and to pick the two items that affected the Alpha 
value the least when omitted. In most cases, the values were very close in terms of what 
contributions they made to Alpha. A single exception was in the knowledge and skills subscale where 
one of the two lowest contributors to the subscale Alpha had the word “knowledge” in the stem. 
Since that was the name of the subscale, that item was retained and another item (very close in its 
contribution to Alpha) was omitted. The judgment was made by the two authors and jointly agreed 
upon. 

The third step in the instrument reduction phase was to recompute the new Alpha reliabilites 
for each of the shortened subscales and the total impact instrument. Table 2 presents the results of 
this recomputation of the Alpha reliabilities for the seven 5-item subscales and the total instrument. 
For the reader’s convenience, the Alpha reliabilities for each of the prior three years’ field tests are 
provided for comparison purposes. Inspection of Table 2 showed that the effects on the Alphas of 
reducing the subscales were very minor. The largest drop in the Alpha value from 1994 (the year 



Table 2 



Impact Instrument Alpha Reliabilities for 
Three Field Tests and Reduced Version 







Field Test Version 


Reduced Version 




Subscale/Scale 


No. 


Alpha Reliability Coefficient 


No. 


Alpha 

Coefficient 




Name and 


of 


Field 

Test 


Field 

Test 


Field 

Test 


of 






Symbol 


Items 


Items 


For This Study 








#1 


#2 


#3 






1 . 


Overall Satisfaction - S, 


7 


.95 


.94 


.94 


5 


.93 


2 . 


Knowledge and/or Skills 

-s 2 


7 


.95 


.95 


.96 


5 


.94 


3. 


Information Sharing - S, 


7 


.94 


.94 


.94 


5 


.92 


4. 


Planning - S 4 


7 


.96 


.95 


.95 


5 


.94 


5. 


Use in Implementation - 

s 5 


7 


.98 


.96 


.97 


5 


.96 


6 . 


Indirect Benefits - S 6 


7 


.94 


.94 


.95 


5 


.94 


7. 


Secondary Clients’ Use 
in Implementation - S 7 


7 


.98 


.95 


.96 


5 


.96 


Total Impact Instrument 


49 * 


.99 


.98 


.99 


35 


.98 



* Does not include the single veracity-check item. 
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with the largest sample) was .02, and one subscale (#7) did not drop at all. The Alpha reliability of 
the total impact instrument dropped from .99 to .98 — still very high. Therefore, reducing the AEL 
impact instrument from 50 items to 35 items yielded virtually no negative results on the Alpha 
reliabilities and, on the positive side, produced an instrument that was more convenient for 
respondents. 
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NORMING PROCESS 



The data used for the norming process consisted of the responses to the 35 items, five per 
subscale, that remained after the 50-item questionnaire was reduced. Over the three years of the 
survey, there were 672 individuals who had responded to all items of the survey. As with any survey, 
respondents will omit items for various reasons. An initial count of 822 individuals had responded 
to at least 26 of the original 50 items, but 1 50 of these had omitted at least one of the 35 items to be 
used for norming, thus reducing the norm group to 672. A check of the percentage distribution of 
the five role groups of interest in the two groups showed the following percentages: 



These percentages within a category across the two groups were very close; the difference 
between any corresponding percentages did not exceed 1%. So the omission of items was not related 
to the role of the respondent. 

Description of the Norm Group 

The 672 individuals in the normative group were employed by a variety of agencies and 
organizations and, of course, most of these were involved in education. The distribution of employers 



672 cases group 822 cases group 



Teacher 

Principal/ Assistant Principal 
Central Office Staff 
Higher Education— Professor 
Higher Education— Admiminstrator 



40.0 
9.2 

10.7 

10.1 
7.0 



40.0 

8.6 

9.9 
9.7 

7.9 



follows. 



Local education agency 
State education agency 
Higher education institution 
Policy-oriented agency 
Professional education agency 
OERI, Lab, or R&D agency 
U.S. Department of Education 
Attended as a parent 
Community organization 
Other 

Four respondents did not indicate an employer and, if employed, they most likely fit the 
“Other” category. As expected, the concentration of respondents, over 84%, were in local education 
agencies, colleges and universities, and state education agencies. 

Five subgroups with substantial numbers were identified using the professional role of the 

respondent. These five role groups and their numbers are 
Teacher 

Principal/ Assistant Principal 
Central Office Staff 
Higher Education-Professor 
Higher Education-Administrator 

These subgroups were of primary interest for comparison to the normative group. Those 

employed by the state education agencies viewed themselves as having a variety of roles and 

apparently many of these individuals selected “Other” for their roles. 

The respondents were quite evenly distributed across the four states served by AEL. West 

Virginia had the largest number with 1 89 or about 28%, and Tennessee had the smallest number with 

134 or about 20%. Thirty respondents lived and/or worked in other states, primarily respondents of 

workshops that had been conducted in Georgia and Indiana. 



269 

62 

72 

68 



404 

49 

114 

12 

16 

1 

1 

2 

33 

36 



60.1% 

7.3 
17.0 

1.8 

2.4 
.1 
.1 
.3 

4.9 

5.4 
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Norm Group Statistics 

The focus of interest was on the seven impact category subscales identified earlier--S lt 
measuring overall satisfaction, through S 7 , secondary clients’ use. The means, standard deviations, 
and reliability coefficients for the scales are provided in Table 3 for all 672 respondents who became 
the “norm group.” The mean for the total scale, all 35 items, was 123.56, and the total score 
distribution had a standard deviation of 33.65. The Alpha internal consistency reliability of the total 
scale was .98. 

Scores on each of the subscales could range from a minimum of 5 to a maximum of 25. The 
distribution for each subscale was converted to percentiles and these percentiles became the basis for 
comparisons among the various subgroups of the norm group. The total score distribution also was 
converted to percentiles, although it was not used in any of the comparisons reported here. 

Positioning of the Norm Group on the Impact Category Subscales 

Although the primary interest was on comparisons of subgroups in the norm group, the 
positions of the entire group on the scales of measurement also are of interest because such positions 
reflect the perceived impact of AEL’s products and services. All of the items were 5-point Likert 
scales, on which 1 indicated “Not at all” and 5 indicated Much, Very, or Many. The midpoint, 3, 
was defined as “Somewhat.” Thus the minimum score for a subscale was 5 and the maximum score 
was 25. The subscale means, generally, followed a decreasing pattern as the type of impact became 
more specific and, in the case of S 7 , one step removed from the respondent. The means for S 4 (use 
in planning) and S 5 (use in implementation) did not follow the decreasing pattern. The means had 
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Table 3 



Means and Standard Deviations for the Norm Group 
by Impact Category Subscale 



Subscale Symbol/ 
Name 


Mean* 


Standard 

Deviation 


S, - Overall Satisfaction 


20.74 


4.10 


S 2 - Knowledge and/or Skills 


19.69 


4.78 


S 3 - Information Sharing 


17.47 


5.60 


S 4 - Planning 


16.80 


5.73 


S 5 - Use in Implementation 


16.99 


5.69 


S 6 - Indirect Benefits 


16.64 


5.71 


S 7 - Secondary Clients’ Use 


15.23 


6.05 


in Implementation 







‘Possible scores were from 5 to 25. 
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only a .19 difference with S 5 having the greater mean. Standard deviations for the subscale 
distributions were quite homogenous. As the mean decreased, there was a tendency for the standard 
deviation to increase, undoubtedly due in part to means positioned closer to the center of the 
measurement scale, providing more opportunity for positioning farther from the mean on the high end 
of the subscale. 

A total subscale score around 20 indicates an average item score around 4, midway between 
“Somewhat” and “Much, Very, or Many.” Even a total subscale score of 1 5 indicates an overall 
perception of “Somewhat” impact. So, it can be concluded that AEL’s products and services had 
substantial impact as perceived by the norm group. Overall satisfaction was the highest, as might be 
expected. But even for secondary clients’ use, there was some impact. 
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EMPLOYER AND PROFESSIONAL ROLE GROUPS RESULTS 



The procedure for comparing any two or more subgroups of the norm group was to develop 
profiles of the seven impact category means as to their positions in the percentile distribution of the 
norm group. The focus of interest was on the relative positions. These are presented below for each 
subgroup of interest, then a composite profile is given which includes the means of all subgroups 
being compared. 

The initial comparison below is for individuals employed by three different agencies or 
institutions: the local education agency, the state education agency, and an institution of higher 
education. The second comparison provides a more detailed breakdown of those employed by local 
education agencies and colleges/universities. For the former, the respondents were divided into 
teachers, principals/assistant principals, and central office staff. For the latter, the division was 
between professors and administrators (deans, department chairpersons, etc.). The professional role 
provided a “finer cut” of the respondents in local education agencies and colleges/universities than 
did the type of employer. But the first comparison was made to include those employed by state 
education agencies, a substantial portion of those serviced by AEL. 

The profiles for those employed by local education agencies, state education agencies, and 
institutions of higher education are given respectively in Figures 1, 2, and 3; Figure 4 contains the 
composite for these three subgroups. The profiles for those employed by local education agencies 
and state education agencies are in a relatively tight band of about 10 percentile points for most 
subscales, and for the final three subscales come quite close together. The widest difference is in S 4 
(planning) for which state education agencies had the highest impact. This result was reasonable in 
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Figure 1. Profile of Means Across the AEL Impact Category Subscales 
for Those Clients Employed by Local Education Agencies 
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Percentile of the Norm Group 




Figure 2. 
for 



Profile of Means Across the AEL Impact Category Subscales 
Those Clients Employed by State Education Agencies 
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Figure 3. Profile of Means Across the AEL Impact Category Subscales 
for Those Clients Employed by Institutions of Higher Education 
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Figure 4. Compoeite Profile of Means Across the AEL Impact Category 
Subscales for Three Employer Subgroups 
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that the local education agency subgroup had a majority of teachers who, as will be seen in the role 
profiles, positioned low on planning. State education agency employees probably are involved in 
more planning, at least at a more formal level, than teachers. 

Those in colleges and universities consistently positioned lowest in the profiles of these three 
subgroups. Overall satisfaction was highest along with planning impact, but even these were five 
percentiles below the lowest mean o** the other two subgroups. These results show AEL s 
consistently greater perceived impact with those in local education agencies and state education 
agencies than those in colleges and universities. This is not to say that the latter are not impacted by 
AEL’s products and services. The lowest mean on a subscale for those in colleges and universities 
was 13.4, still in the “Somewhat” impact range, and this mean was on S 7 . Except for the teacher 
education students, it is not apparent who are perceived by professors and college administrators as 
secondary clients receiving AEL products and services. There may be some higher education 
individuals working with educators in the elementary and secondary schools that view such educators 
as being impacted by AEL through the universities and colleges, but such cooperative activity is not 
extensive, at least not in a formal manner. 

The identification of subgroups by professional role provides more detail for those working 
in local education agencies and institutions of higher education. The majority of those in local 
education agencies were teachers, with over 66%, but there were substantial numbers of 
principals/assistant principals and central office staff as well. The higher education respondents were 
split about 60-40, with the greater percentage being professors. The profiles for these five 
professional role subgroups are given in Figures 5-9, with the composite profile in Figure 10. 
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Figure 5. Profile of Means Across the AEL Impact Category Subscales 

for Teachers 
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Figure 6. Profile of Means Across the AEL Impact Category Subscales 
for Principals/Assistant Principals 
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Figure 7. Profile of Means Across the AEL Impact Category Subscales 

for LEA Central Office Staff 
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Figure 8. Profile of Meant Acrott the AEL Impact Category Subscales 
for College/University Professors 
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Figure 9. Profile of Means Across the AEL Impact Category Subscales 
for Higher Education Administrators 
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Figure 10. Composite Profile of Means Across the AEL Impact Category Subscales 

for Professional Role Subgroups 
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The composite profile of Figure 10 shows some striking differences in perceived impact 
among the professional roles. Principals and assistant principals consistently have the perceptions 
of most impact and higher education administrators consistently have the perceptions of least impact. 
These two subgroups were as much as 38 percentile points apart. These differences appeared on S 2 
(the knowledge and skills subscale) and S 5 (the use in implementation subscale). 

The reliability of the 35 items measuring impact and the seven impact category subscales has 
been demonstrated conclusively. The items were developed through a process that provided face 
validity. So assuming the validity of the subscales, the norming study provides empirical results 
indicative of the extent and manner that AEL products and services impact the various groups of 
clients served by AEL. Based on these results, conclusions and recommendations can be made about 

AEL’s impact. 
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CONCLUSIONS AND RECOMMENDATIONS 



The results of the impact study showed that the various client groups are impacted differently 
across the seven impact categories. Overall, the profiles of the subscale scores, for both employer 
and professional role, reflect AEL’s emphasis on delivering programs to elementary and secondary 
schools. Profiles indicate that impact will occur where effort is expended. The norming process 
proved valuable in this respect. One conclusion, not dependent on the norming process, but one that 
should be emphasized, is the magnitude of AEL’s impact through its products and services. This 
impact is very substantial as evidenced by the positioning of means on the subscales. 

The relatively high positioning of the principals on the subscales may be due to a combination 

of factors, but two likely factors come to mind: 

(1) AEL’s programs and projects are such that many involve principals as one of the 
participants. 

(2) Principals are in a somewhat unique position as conduits for many AEL-sponsored 
activities, projects, etc. 

Generally, as facilitators or conduits, principals would feel involvement in sharing, 
implementation, planning, and receiving indirect benefits. The involvement would generate a feeling 
of gaining knowledge and skills, and correspondingly a greater feeling of overall satisfaction. 
Teachers, on the other hand, also would have a feeling of knowledge and skills gained, but would not 
feel as strongly impacted by sharing and planning, or even implementation. Central office staff 
perceive themselves as being impacted through planning and sharing and slightly less through 
implementation. Also, they were high on secondary clients’ use in implementation, most likely 
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because of their leadership positions in the school systems. They see the results of their efforts 
transferred to teachers and other professionals in the school systems. 

The fact that both professors and college and university administrators were low on gaining 
knowledge and skills is not surprising. Undoubtedly, they perceive other sources in addition to AEL 
as providing knowledge and, to the extent that it occurs, skills. University personnel have direct 
access to numerous retrieval systems, etc., through the university library and these sources address 
extensively new developments based on research. Most educators have access to such sources, but 
not as readily as those in universities. 

Another factor that may be operating is that personnel in colleges and universities are impacted 
by other forces that take precedence over the types of activities promoted by AEL. There are 
separate “agendas” in colleges and universities that are not as compatible with AEL activities as the 
needs of the elementary and secondary schools. For example, collaboration with elementary and 
secondary schools is not frowned upon in colleges and universities, but it is not one that is particularly 
favored in the rewards system. Scholarly publishing generally has a much higher priority. Also, 
implementing innovative approaches to teaching is more difficult at the college level than in 
elementary and secondary schools. 

However, it should not be concluded that those in colleges and universities are being ignored. 
Indeed, one AEL program, Colleges and Schools, directly addresses the needs of college and 
university personnel; and many of the other programs involve professors and higher education 
administrators, or at least touch upon some of their activities. 

State education agency personnel had profiles similar to principals/assistant principals and 
central office staff in local education agencies. These groups were consistently high across the 
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subscales, but especially so in planning and clients’ use in implementation. These patterns reflect 
AEL’s impact upon those in positions of educational leadership. State education agency personnel 
provide leadership at a different level than those in local education agencies. But it is to AEL’s credit 
that it consistently impacts educational leaders across these levels. 

As an overall recommendation, the results and conclusions from the impact study would imply 
that AEL “should continue doing what it has been doing.” Certainly, a major objective of any 
Regional Educational Laboratory is to impact the activities of educational leaders. AEL has shown 
its impact upon principals, school system central office staff, and state education agency personnel. 
In whatever manner AEL’s programs impact these leaders, and undoubtedly this is done in many 

ways across the nine programs, this should be continued. 

Certainly, no one would suggest diminishing the impact at the school, local education agency, 
and state education agency levels. The impact at the college and university level might be increased, 
if it could be done without decreasing the efforts at the other levels. Possibly, impact at the college 
and university level is at a maximum given all the conditions and factors that exist. Or possibly, some 
changes in addressing the needs of college and university personnel might enhance the impact. 

The norming process has provided valuable information about the impact of AEL’s products 
and services, as reflected in a substantial client base. The results show the extent of impact in an 
absolute sense, and they show the different patterns of impact among the subgroups in AEL’s client 
population. The former information is important as measures of impact for the seven specified 
categories. The latter is important for differentiating impact among the subgroups and may have 
implications for program emphasis and possible adjustment. 
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APPENDIX 



AEL SEDCAR Standards Checklist 


The Standards tor Educational Data □ study group report □ Research report 

Collection and Reporting (1991) □ p, e y test report Impact study report 

were used in the development of n ».. • „ r~i 

this (check one): D Migrant final report □ Other 

□ Implementation report 

The SEDCAR Standards were consulted and used as indicated in the table below (check or mark as appropriate): 


SEDCAR 

' Standard Number 
and 

Descriptor 


The Standard was 
deemed applicable; and, 
to the extent feasible, 
was taken into account.* 


The Standard was 
deemed applicable; 
but could not be taken 
into account. 


The Standard 
was not 
deemed 
applicable. 


Exception 
was taken 
to the 
Standard. 


1 .1 Creatir j an Infrastructure to Manage 
Date Collection Activities 






XXXX 

.Secondary 

Analysis) 




1 .2 Justifying Data Collection Activities 






KXXX (Secon- 
dary Analysi 


s) 


1 .3 Fosterinr Commitment of all Participants 






XXXX (Secon- 
dary Analysi 


s) 


1 .4 Creating an Appropriate Management 
Process 


xxxx 








2.1 Formulating and Refining Study Ques- 
tions 


xxxx 








2.2 Choosing the Data Collection Methods 






XXXX (Secon- 
dary Analysi 


s) 


2.3 Developing a Sampling Plan 






XXXX (Secon- 
dary Analysi 


s) 


2.4 Assessing the Value of Obtainable Data 


xxxx 








2.5 Transforming Study Question Concepts 
into Measures 


xxxx 








2.6 Designing the Data Collection Instru- 
ment 






XXXX 

(Secondary 

Analysis) 




2.7 Minimizing Total Study Error (Sampling 
and Nonsampling) 






XXXX 

(Secondary 

Analysis) 




2.8 Reviewing and Pretesting Data Collec- 
tion Instruments, Forms, and Procedures 






XXXX 

(Secondary 

Analysis) 




2.9 Preparing a Written Design 


xxxx 








3.1 Preparing for Data Collection 






XXXX (Secon- 
dary Analys; 


s) 


3.2 Selecting and Training Data Collection 
Staff 






XXXX 




3.3 Ethical Treatment of Data Providers 


xxxx 








3.4 Minimizing Burden and Nonresponse 






XXXX (Secon 
dary Analys 


:.s) 


3.5 Implementing Data Collection Quality 
Control Procedures 


xxxx 









r column headings from Evaluation Standards. 
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SEDCAR 
Standard Number 
and 

Descriptor 


The Standard was 
deemed applicable; and, 
to the extent feasible, 
was taken into account.* 


The Standard was 
deemed applicable; 
but could not be taken 
into account. 


The Standard 
was not 
deemed 

applicable. 


Exception 
was taken 
to the 
Standard. 


3.6 


Documenting Data Collections 


xxxx 








4.! 


Planning Systems Requirements 


xxxx 








4.2 


Designing Data Processing Systems 


xxxx 








4.3 


Developing Data Processing Systems 


xxxx 








4.4 


Testing Data Processing Systems 






XXXX (Used 
Standard 




4.5 


Planning for Data Preparation 


xxxx 




Software] 




4.6 


Preparing Data for Processing and 
Analysis 


xxxx 








4.7 


Maintaining Programs and Data Files 


xxxx 








4.8 


Documenting Data Processing Activities 


xxxx 








4.9 


Evaluating Data Processing Systems 


xxxx 








5.1 


Preparing an Analysis Plan 


xxxx 








5.2 


Developing Analysis Variables 


xxxx 








5.3 


Applying Appropriate Weights 






XXXX 




5.4 


Estimating Sampling and Nonsampling 
Errors 






xxxx 




5.5 


Determining Statistical Significance 






xxxx 




6.1 


Presenting Findings 


xxxx 








6.2 


Reviewing the Report 


xxxx 








6.3 


Releasing Data 


xxxx 








6.4 


Disseminating Data 


xxxx 








6.5 


Preparing Documentation and Technical 
Reports 


xxxx 
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